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ABSTRACT 

The critical reading of research involves: (1) 
comprehension, (2) evaluation, and (3) application. A study examined 
six recently published textbooks to determine to what extent they 
attempt to help students learn to apply educational research; these 
texts were specifically designed for "consumers" of research (i.e., 
critical readers of research) as opposed to those who do research. In 
general, these texts emphasized the content needed to comprehend 
research and the evaluation of research as an activity separate from 
intended use. That is, all texts emphasized the comprehension and 
evaluation of research as research, with external validity being the 
main concern that would fall under application. Consistent with the 
positivist outlook assumed in most of these books, the concerns and 
functional knowledge of the practitioner was not much in evidence in 
most of these books beyond the introductions. The results of the 
study, therefore, were disappointing in that they revealed that there 
was vry little attention paid to research applications in these 
books. The assumption seems to be that once the reader has 
comprehended the research and evaluated it, the implications will be 
evident. Operating within a quantitative framework, none of the books 
specifically dealt with the problems of moving from a statistically 
defined tendency within a population to making decisions about a 
non-random sample of that population. There was little effort to 
explain validity as a concept that changes both with the nature of 
the thing being evaluated and the pe-son doing the evaluating. 
(Contains 20 references.) (TB) 
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Applying Research 

Abstract 

The critical reading of research involves comprehension, evaluation, and application. Using 
application as a context, we examined six educational research texts which were explicitly written 
for consumers of research. In general, the texts did not emphasize application and assumed a 
limited view of what application might include. 
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Applying Research 3 
Applying Research: An Analysis of Texts for Consumers of Research 

The teacher who wants to determine if cooperative learning works in his first grade 
classroom and engages in action research or the testing specialist who designs a study to test her 
hypothesis on the interaction between speededne^s and predictive validity have little need to 
wondi;i about the utility of their research in terms of their own personal needs. Nor does the team 
thai puts together a series of surveys on behalf of a school board. In these cases, particular 
questions have been asked and the research designed to provide information viewed as specifically 
useful to the individuals who are going to make decisions and set policies. However, in all of these 
cases, there is a possibility that the report of the research will be published. The readers of these 
research reports will be operating in different contexts than those who performed or requested the 
original studies. Although the levels of sophistication will vary, these readers are consumers of 
research inlbrmation. 

It was in this latter context that D. C. Phillips (1980) asked, "What do the researcher and 
the practitioner have to offer each other?" His conclusion was that there wasn't very much. Few in 
the university educational community today, with the increased emphasis on practitioners as 
researchers and research partners, would agree in terms of what practitioners have to offer. Nor 
would it appear that most educators at the university level would agree that researchers have little 
to offer, although this was the point Phillips argued most earnestly, since learning to critically read 
research reports is part of the meat of most graduate programs in the helping professions/social 
sciences. Counselors, teachers, school, administrators, sociologists, nurses, and social workers 
may have programs with expectations that vary in many respects, but all seem to support the idea 
that graduates of advanced studies at the very least must be competent consumers of research. 

What does it mean to "consume" research. While several interesting images come to mind 
(If there are enough consumers, will all the research be used up?), the intent seems to involve the 
critical reading of research. We started with a three component model for critical reading of text. 
When people read critically, they must understand the intent of the authoF (comprehension). 
Furthermore, they must make judgments of worth concerning what the author is saying 
(evaluation). The content, along with judgments of worth must then be considered in terms of 
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significance for decision making processes and practice within contexts which are characteristic of 
the reader (appHcation). Similar or compatible schemes are common in subject area and critical 
reading models, and this one had formerly been used by us to describe the critical reading of 
reviews (Steinley, 1991; Steinley, Erion & Romereim, 1993). Of particular interest to us was the 
third category, application. If consuming means reading research with application in mind, what 
does it mean to apply research? 

Phillips (1980) spoke of researchers coming up with "some solid findings applying to ail 
children, or to some important subsets of children" (pp. 18-19). He argued that 1) research results 
do not come in a form which simply indicate how they should be generalized, but rather links have 
to be established between the world of practice and the theory, 2) people with different sets of 
linking premises and assumptions will see the same set of results as having different implications, 
and 3) the premises of researchers often are not made clear (are suppressed'* so that the conclusions 
which researchers draw from their research ought to be treated as suspect. A special case of 
suppressed assumptions which Phillips does not discuss is the assumption that the course of action 
which can be argued as most reasonable for a population is most rejisonable for a subset of the 
population, although this can also be approached as a a non-trivial problem in statistics (how likely 
is a tendency in a population to be manifested in a non-random sample of that population?). 

Phillips (1980) seemed to imply that one must generalize from research results to use them 
and this emphasis on generalizing as a necessary part of applying research occupied Firestone 
(1993) as well. In the context of defending the utility of qualitative research, Firestone argued that 
three types of generalization of research must be considered. The most often discussed (sample to 
population) is the strength of quantitative research, not qualitative research. However, research 
results can be generali^e<^ to another case (case to case) or a theory (case to theory). 

Can ^md Kenimis ( 1986), writing in the context of action research, organize educational 
research as being in one of three traditions, a natural science or positi /ist approach, an interpretivist 
approach, and a critical approach, each with a somewhat different purpose for research. In the 
positivist view, research leads to theory which in turn allows prediction or what Livingston and 
Castle ( 1989) characterize as explanation and direction. The interpretivist view serves to "deepen 
insight and enliven commitment," acting indirectly to to influence action "through the mediation of 
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the critical reflection of individual actors" (Carr & Kemmis, 1986, p. 93). The approach to 
research in a critical social science draws heavily on Habermas and is one in which the researcher 
is seen as undertaking an actions with social and political significance with action research 
providing a means to unify theorj' and action. 

Huberman (1990) indicated the effects that research might have as including cognitive or 
affective changes in practitioners (conceptual use) and changes in practice (instrumental use). 
Huberman also indicated that research results could be characterized by the extent and durability of 
their use, as well as the degree to which they lead to transformation. Secondary effects might 
include linkages between organizations and the individuals in those organizations, as well as 
"spillover," organizational, and career effects. 

Livingston and Castle (1989) describe five ways in which teachers might use research. The 
first, which they call application (choice of "use" or "application" seems somewhat arbitrary), is 
use of educational research in the tradition of What Works (United States Department of Education, 
1986). Educational research leads to "big principles" which provide answers or directions for 
practice. The second, related to the first, is justification or validation for a course of action or 
decision. Research can also lead to contemplation and delil eration, similar to the aim of 
interpretivist research (Ca.T and Kemmis, 1986). Transformation is part of ihe critical social 
research concept of Carr and Kemmis (1986). Research can lead to a critical examination of the 
assumptions underiying modes of practice. Lastly, practitioners can produce research to guide and 
examine their own behaviors. 

There are, then, a variety of ways to approach what is meant by application of research and 
a number of factors which might be considered. When we started to try to tease out the 
relationships between comprehension, evaluation, and application, we thought of research 
application as existing along a continuum. At one end was immediate action based on research 
results. Having read that there is little difference in test results when classrooms have 17 students 
or 23 students, a district ups classroom size from 15 to 20. Since praise has been seen as useful in 
classroom situations, one of the topics for inservice days will be the use of praise in the classroom. 
At the other end of the continuum was a more conceptual or interpretive use of research. The 
researcher whose study is directed at clarifying a point in theory and the teacher who reads a 
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review of research on the grading of essay questions may both simply change schema or 
understanding, with no immediate changes in behavior. We have added to this continuum an 
understanding that the nature of generalizing can be variously conceived, depending in part on the 
philosophical position taken on the nature and uses of knowledge. 

The application of a given research study is not necessarily direct, then. When one applies 
research, one starts with the understanding of the research report and one's evaluation of the 
research in light of the situation (contextual factors generally seen as relevant to external validity) 
and ones previous understanding. It is ones understanding (perhaps changed by the interaction 
with an particular research report) that is ultimately applied. 

Research in this model is understood, in ways which are situational to the reader. The 
process of critically reading involves an understanding that takes into account the intent of the 
researcher, the accumulated knowledge of the reader, and the intent of the reader. The evaluation of 
the research (judgments concerning its validity) is analogous to the validation arguments for a test 
where the intended use as well as the characteristics of the test itself must be considered. 
Recognizing this, we looked at Messick (1989) for an overall framework for placing the concepts 
and skills involved in critically looking at research (see Figure 1). Messick used a broad definition 
of testing in his argument, "any means of observing or documenting consistent behaviors or 
attributes" (p. 5). This represents a subset of the activities carried out in research and we suggest 
that Messick' s argument is generalizable to all forms of research in the broad sense. 

In the general sense, the categories in the first row (the evidential basis) include the familiar 
constructs of internal validity and generalizability. It is in Messick' s consequential bases for 
validity that the evaluation process is expanded beyond that generally suggested in texts on 
research. All research involves value-laden choices. Choice of phenomena to study, labels, 
methodology, and interpretations are value-laden activities common to any research endeavor with 
semantic, rhetorical, esthetic, and ethical dimensions. 

Research is a puiposeful activity. Results are reported and interpreted to accomplish goals 
described by the researcher. Any application may have expected consequences and will very likely 
have unexpected consequences, although it is our impression that the social consequences of 
research is most often mentioned when it is judged that there are none. 
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Messick's matrix was specifically aimed at a unitary concept of test validity. Our purpose 
was a unitary concept for evaluating research which included the possible uses of the research. The 
judgments made in research interpretation and use involve comprehension. Research interpretation 
includes internal validity as well as considering value implications. The generalizability of research 
operates in at least several dimensions and is dependent on the intended use. Judgments of use 
should include likely consequences of that use. 





Research Interpretation 


Research Use 


Evidential 
Basis 


Research qua Research 
(RR) 


RR 

+ 

Relevance/Utility (RU) 


Consequential 
Basis 


RR 
+ 

Value Implications (VI) 


RR + RU + VI 
+ 

Social Consequences 



Figure 1. Framework for identification of processes in critical reading of research (based 
on Messick, 1989) 

Having arrived at a fairly broad view of what application might involve, we decided to look 
at how this is handled in textbooks with the belief that textbooks aimed at consumers of educational 
research are intended to provide a means for practitioners to become critical readers of research. If, 
as we have supposed, critical reading involves attending to comprehension, evaluation, and 
application, how is this reflected in these textbooks? More specifically, how and to what degree is 
application of research addressed in texts intended for instruction of consumers of educational 
research in terms of differing perspectives of what it means to apply research? 

In order to explore this question, we selected six relatively recent texts (see Table 1) which 
are distinguished from the many other educational research texts that have appeared over the last 
six years in that they are specifically intended for consumers as opposed to those who will do 
research. We both read each text independently looking for instances of application from any of the 
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perspectives suggested above. We particularly considered philosophical perspective as indicated in 
Carr and Kemmis (1986). From Livingston and Castie (1989) we took the list of possible uses of 
research which seemed to go beyond Huberman's conceptual and instrumental categories. We also 
considered the discussion of generalizability, m particular noting whether it went beyond the 
concept of external validity. We used the modification of Messick's matrix for making broad 
generalizations since it quickly became evident that none of the authors had had something like this 



in mind when they wrote their texts. 



Table 1 

Texts Analvzed 


Author(s) 


Publication Title 
Date 


Comments 


Black 


1993 


Evaluating Social Science 
Research 


After an overview of research, 
organized around different aspects 
of research to be evaluated. 


Borg, Gall, & Gall 


1993 


Applying Educational 
Research: A Practical Guide 


Definitely practitioner oriented. 
Types of research. Includes 
forms for evaluation quantitative 
and qualitative research. 


Hittleman & oimon 


1992 


Interpreting Educational 
Research: An Introduction for 
Consumers of Research 


Organizes evaluation of research by 
section of research report with 
questions for each section, 
differentiated by type of research. 


Jaeger (ed.) 


1989 


Complementary Methods 
for consumers. 


AERA publication with chapters 
on types of research by respected 
names in field. Not intended solely 


McMillan 


1992 


Educational Research 


Chapters on types of research 
follow philosophical underpinnings 
of researched. Criteria for 
evaluation generally ignore 
practitioner's situation. Includes 
evaluation questions for 
quantitative, historical, and 
qualitative research. 


Viera & Pollock 


i988 


Reading Educational Research 


Organized on broad separation of 
quantitative (more extensive) and 
qualitative research with emphasis 
on science as theorv building. 
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In general, these texts emphasized the content needed to comprehend research and the 
evaluation of research as an activity separate from any intended use. That is, all texts emphasized 
the comprehension and evaluation of research as research, with external validity being the main 
concern that would fail under application. Other than a generally positivist outlook (Carr & 
Kemmis, 1986), not tnie of some of the chapters in Jaeger ( 1989) (eg. Wolcott's chapter on 
ethnography), the textbooks we examined did not lend themselves to many other facile 
generalizations. Consistent with the generally positivist outlook, but somewhat surprising in 
books written specifically for practitioners, the concerns and functional knowledge of the 
practitioner was not much in evident in most of these books beyond the introductions. For 
example, although McMillan ( 1992) explicitly recognizes the importance of a situational 
understanding of research, he described the purpose of science as being to describe, predict, 
control, and provide explanations, with the main purpose being sound explanations. Hittleman and 
Simon ( 1992) characterize research consumers as answering "educationally related questions [by] 
reading an applying [the results of] research producers" (p. 2). "Teachers at all levels continually 
make decision about instructional activities... They base these decisions on their experiences, other 
teachers' experiences, and their understanding of accumulated knowledge about education" (p. 1), 
but their emphasis in evaluating research was in terms of examining methodology. In Jaeger, 
Shulman (1989), discussing generaiizability, noted the problems caused by the usual lack of 
random samples, but ignored the problem in moving from a population back to some segment of 
that population. He used the metaphor of an inl'erential bridge (based on a 1956 paper by Cornfield 
and Tukey) for generalizing from a sample to a sample and extended it to generalizing across 
settings and tasks. The task of the reader of research is to determine if inferential bridges can be 
built, noting that different research methods "carry with them different ways of asking questions 
and often different commitments to educational and social ideologies" (pp. 11-12). 

When the knowledge of the practitioner was noted, it tended to be devalued. Borg, Gall, 
and Gall (1993), for example, discussed four approaches to making educational decisions: 
personal experience, expert advice, searches of the research literature, and conducting action 
research. These were apparently seen as disparate approaches. The third approach was 
recommended after a discussion the limited nature of personal experience and a suggestion that 
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experts be primarily used to "lcx:ate relevant rCvSearch literature rather than relying entirely on the 
conclusions that they have reached on the basis of this literature" (p. 10) . Action research was 
suggested if 

...a literature search does not produce a satisfactory solution to your problem ...[since] 
...action research is a far better basis for decision making than personal expe rience. Most 
of the discoveries and progress made by Western civilization during the past two hundred 
years have been achieved by using the scientific method to attack problems and questions. 
Action research employs the scientific method to solve the problems of the educational 
practitioner . (p. 11) 

In a discussion of the need to consider the academic field in evaluation of research, using the 

example of addiction, Viera and Pollack (1988) stated: 

Interestingly, some of the most heuristically useful work in this field is being done not by 
professional researchers but by practitioners. Practitioners, whether they conceptualize 
themselves in this way or not, are in a very real sense doing qualitative research: 
empirically immersing themselves in the topic. They can thus be more likely to discover 
previously unsuspectal patterns than are researchers, especially if the researchers have a 
quantitative bias and only collect data on variables they have previously defined as relevant. 
These practitioners' qualitative data were not collected to test hypotheses; their suggestions 
about causal relationships in addiction are only that-suggestions. And their samples are very 
biased; generalizability of their observations is thus very limited." (p. 232) 

On the other hand, McMillan ( 1992) cautions readers to "Keep a balanced perspective about the 

relative contributions of research and professional judgment" (p. xvi). 

As indicated above, all texts did address general izabity. With the exception of Viera and 

Pollack ( 1 988) and Jaeger ( 1989), this was overwhelmingly a discussion on external validity with 

Black ( 1993) restricting his discussion to the "representativeness" of the sample. Viera and Pollack 

discussed at length what Firestone (1993) described as generalizing to theory. Given the 

multiplicity of chapter authors and approaches it is not surprising that different viewpoints on 

generalizability were expressed in Jaeger. Wolcott (1989), for example, tells the reader "I don't 

believe that educational research of any type has yet had great impact on educational practice, and 

descriptive research portraying how things really are does not seem to capture the imagination of 

those impatient to make them different" (p. 203). He adds that 

...ethnography does not point out the lessons to be gained or the action that should be 
taken. Worst still, anyone who takes the time to read a descriptive account will probably 
realize that the complexity of the setting or problem at hand has been increased rather than 
decreased, (p. 203) 
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Porter ( 1989), as an example of the most common approach, discusses in his chapter on 

comparative experiments generaliaing across subjects, experimental conditions, and outcomes. 

"One way to think about external validity is to imagine a particular experiment and then attempt to 

identify situations in which you believe the results might not apply" (p. 405). Borg et al. directly 

addressed generalizing from case to case and explained that 

...results may have high external validity for some local settings and low external validity 
for others, depending on the similarity between the research conditions and local 
conditions. Thus, potential users must compare the experimental conditions with the local 
conditions to which they want to generalize the results and make a decision about how well 
the findings are likely to apply, (p. 303) 

They go on to say that in order to apply research, one must see if local population is like the 
research sample. Their discussion also raised the issue of interactions (do results apply to some 
members of population, but not others) and the degree to which settings are similar. 

The results of our study were less than satisfying in that there was very little attention paid 
to application. The implication seems to be that once one has comprehended the research and 
evaluated it, the implications will be evident. Operating within a quantitative framework, none of 
the books specifically dealt with the problems of moving from a statistically defined tendency 
within a population to maldng decisions about a non-random sample of that population. In general, 
there was little effort to explain validity as a concept which changes both with the nature of that 
being evaluated and the evaluator. Ctespite the inclusion of qualitative research as one form of 
educational research, the interpretivist and critical theory approaches to understanding educational 
phenomena with their implications for practice were not generally in evidence. Nor was a general 
discussion on philosophy of science with implications for comprehension, evaluation, and 
application, although both the McMillan and Jaeger texts explicitly addressed the importance of 
understanding different perspectives on research.. Eichelbergcr ( 1989) provides an example of 
how this might be approached in the first chapter of his textbook for researchers (the last chapter is 
an explicit discussion of using research to help understanding and to make decisions). Both 
chapters would make excellent additions to any text intended for consumers of research. 

Over the last 50 years there have been periodic attempts to see if educators read the 
professional literature. In almost all cases, the findings have indicated that they generally don't 
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(Latham, 1993). The lack of attention to application in texts which are presumably intended to help 
practitioners become people who will understand and use the research may explain, at least in part, 
the perception that reading reading research is not useful. 
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